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Abstract- In  this  paper,  we  introduce  Nonadditive  Information 
Theory  through  the  axiomatic  formulation  of  Tsallis  entropy. 
We  show  that  systems  with  transitions  from  high 
dimensionality  to  few  degrees  of  freedom  are  better  described 
by  nonadditive  formalism.  Such  a  biological  system  is  the  brain 
and  brain  rhythms  is  its  macroscopic  dynamic  trace.  We  will 
show  with  simulations  that  Tsallis  entropy  is  a  powerful 
information  measure,  and  we  present  results  of  brain 
dynamics  analyzed  using  EEG  recordings  from  a  brain  injury 
model. 
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I.  INTRODUCTION 

Nonadditivity,  or  nonextensivity,  is  an  important  concept 
in  the  field  of  information  theory.  A  system  is  nonextensive 
if  it  contains  long-range  interaction,  long-range  memory,  or 
(multi)fractal  structure  [1],[2],[3].  In  such  a  system,  a 
macroscopic  dynamic  quantity  is  not  simply  proportional  to 
the  microscopic  degrees  of  freedom  but  there  are  moments 
where  the  system  unpredictably  looses  its  high 
dimensionality  (described  by  stochastic  process)  and  falls  to 
a  low  dimensional  deterministic  system  (either  linear  or 
not).  Classical  paradigms  in  brain  dynamics  is  the  rise  of  an 
epileptic  crisis  or  the  burst  suppression  EEG  following  the 
resuscitation  from  cardiac  arrest  or  asphyxia  [13], [14].  In 
this  respect,  a  nonextensive  generalization  of  Boltzmann- 
Gibbs  statistical  mechanics  formulated  by  Tsallis  [4]  is 
better  suited  to  describe  such  phenomena. 

II.  NONADDITIVE  INFORMATION  THEORY 

In  the  development  of  the  foundations  of  classical 
information  theory,  Khinchin  [5]  presented  a  mathematically 
rigorous  proof  of  a  uniqueness  theorem  for  the  Shannon 
entropy  based  on  the  additivity  law  for  a  composite  system 
in  terms  of  the  concept  of  conditional  entropy.  Suppose  the 
total  system  can  be  divided  into  two  subsystems,  A  and  B , 
and  let  pij(A,B)  be  the  normalized  joint  probability  of 
finding  A  and  B  in  their  z'th  and  y'th  states,  respectively.  Then 
the  conditional  probability  of  B  given  A  found  in  its  z'th  state 

is  given  by  pq  (fl|  A)  =  /u.  (A,  B)/  pj  (A),  which  leads 
to  the  celebrated  Bayes  multiplication  law 

Pij(A,B)  =  pi(A)pij(B\A),  (1) 


where  p,(A )  is  the  marginal  probability  distribution: 
pt  (A)  =  V  pu  (A,  B) .  It  should  be  noted  that  this  form  of 

factorization  can  always  be  established  in  any  physical 
situation.  The  Shannon  entropy  of  the  composite  system  is 

S(A,  B )  =  -kB  £  Pij  (A,  B)  In  Pij  (A,  B)  \  (2) 

ij 

Combining  (1)  and  (2)  yields 

S(A,B)  =  S(A)  +  S(EtlA),  (3) 

where  S(B |A)  stands  for  the  conditional  entropy  [6]. 

In  the  particular  case  when  A  and  B  are  statistically 
independent,  pIJ(B\A)=pl(B)  and  from  (1)  and  (2)  is  drawn 
out  the  additivity  law  S(A,  B)  =S(A)  +S(B).  We  emphasize 
here  that  there  is  a  natural  correspondence  relation  between 
the  multiplication  law  and  the  additivity  law: 

P.jiAB)  =  Pi(A)Pij(E\A)~S(AB) =S(A)+S(E^A),  (4) 

When  the  above  discussion  is  generalized  to  any  composite 
system  there  are  theoretical  and  experimental  considerations 
where  systems  do  not  obey  to  the  additivity  law  [7].  In  this 
respect,  a  nonextensive  generalization  of  Boltzmann-Gibbs 
statistical  mechanics  formulated  by  Tsallis  [4]  is  better 
suited  to  describe  such  phenomena.  In  this  formalism, 
referred  to  as  nonextensive  statistical  mechanics.  Shannon 
entropy  in  (2)  is  generalized  as  follows: 

59(A,B)  =  T^-{X;  t„(A,B)}  -lj.  (5) 

where  q  is  a  positive  parameter.  This  quantity  converges  to 
the  Shannon  entropy  in  the  limit  q—>  1.  Like  the  Shannon 
entropy,  it  is  nonnegative,  possesses  the  definite  concavity 
for  all  q>0,  and  is  known  to  satisfy  the  generalized  H- 
theorem  .  Nonextensive  statistical  mechanics  has  found  a  lot 
of  physical  applications.  A  standard  discussion  about  the 
nonadditivity  of  the  Tsallis  entropy  Sq(p)  assumes 
factorization  of  the  joint  probability  distribution  in  (1) 

( Py  (A,  B)  =  Pj  (A) Pj  ( B ) ).  Then,  the  Tsallis  entropy  is 
found  to  yield  the  so-called  pseudoadditivity 

sq  (A,  B)  =  Sq  (A)  +  Sq  (B)  +  (1  -  q)Sq  (A)Sq  (B) .  (6) 

Clearly,  the  additivity  holds  if  and  only  if  q—>  1.  However, 
there  is  a  logical  difficulty  in  this  discussion.  As  mentioned 
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above,  Tsallis'  nonextensivity  was  devised  in  order  to  treat  a 
system  with,  for  example,  long-range  interactions.  On  the 
other  hand,  physically,  “dividing  the  total  system  into  the 
subsystems"  implies  that  the  subsystems  are  made  spatially 
separated  in  such  a  way  that  there  is  no  residual  interaction 
or  correlation.  If  the  system  is  governed  by  a  long-range 
interaction,  the  statistical  independence  can  never  be 
realized  by  any  spatial  separation  since  the  influence  of  the 
interaction  persists  at  all  distances.  In  fact,  the  probability 
distribution  in  nonextensive  statistical  mechanics  does  not 
have  a  factorizable  form  even  if  systems  A  and  B  are 
dynamically  independent,  and  therefore  correlation  is 
always  induced  by  nonadditivity  of  statistics  [8]. 

Thus,  it  is  clear  that  the  assumption  of  the  factorized  joint 
probability  distribution  is  not  physically  pertinent  for 
characterizing  the  nonadditivity  of  the  Tsallis  entropy. 
These  considerations  naturally  lead  us  to  the  necessity  of 
defining  the  conditional  entropy  associated  with  the  Tsallis 
entropy. 

To  overcome  the  above  mentioned  logical  difficulty  and 
to  generalize  the  correspondence  relation  in  (4) 
simultaneously,  Santos  [9]  proposed  a  generalization  of 
Shannon’s  theorem  to  Tsallis  entropy,  Hotta  et  Joichi  [10] 
investigated  composability  and  generalized  (Tsallis)  entropy 
and  Abe  [11]  extended  the  work  of  Santos  [9]  generalizing 
the  Khinchin  axioms  for  the  ordinary  information  theory  in  a 
natural  way  to  the  nonextensive  systems. 

Along  the  lines  of  recent  paper  of  Abe  and  Rajagopal 
[12]  we  consider  the  Tsallis  entropy  of  the  conditional 

probability  distribution  pjj(A\B)=  Pij  (A.  B)  /  Pj  (A)  as 


.  S(A,B)-S(A) 

Sa  (B  A)  =  - - - q— . 

9  1  1  +  (1-<7)S,(A) 

From  this,  we  immediately  see  that 

Sq(AB)  =  Sq(A)+Sq$A)+(l-q)Sq(A)Sq$A) 


(7) 

(8) 


which  is  a  natural  nonaddititive  generalization  of  (3)  in 
view  of  pseudoadditivity  in  (6).  Therefore  the 
correspondence  relation  in  (4)  becomes  now 


Pij(A,B)  =  Pi(A)Pij(B\A)~ 
Sq(AB)=Sq(A)+Sq(^A)+a-q)Sq(A)Sq(^A).  (9) 


This  equation  coincides  with  the  (6)  of  pseudoadditivity 
when  the  two  systems  A  and  B  are  independent. 

In  this  way  the  nonadditive  Tsallis  entropy  was 
formulated  according  to  the  Khinchin  axioms  of 
information  theory  and  the  contradiction  between 
dependency  (Bayes  law)  and  long  range  interaction 
(nonadditivity)  is  removed.  Summarizing,  we  have 
established  on  firm  mathematical  grounds,  a  general 
criterion  for  consistent  testing  of  the  independence  between 
random  variables,  which  we  propose  as  a  practical  tool  to 
analyze  the  EEG  recordings.  The  results  depends  upon  the 
entropic  index  q.  It  is  expected  that,  for  every  specific  use, 
better  discrimination  can  be  achieved  with  appropriate 
ranges  of  values  q. 


Fig.  1.  A  synthetic  signal  (top)  and  Shannon  and  Tsallis  entropy  for 
different  q  index  values(^=1.5,3,5).  The  synthetic  signal  is  composed  by: 
(I)  one  realization  of  Gaussian  distribution  with  zero  mean  and  standard 
deviation  equals  1,  (II)  one  realization  of  uniform  distribution  in  the 
interval  [-2,2],  (III)  EEG  recordings  from  baseline  (III)  and  early  recovery 


(IV).  For  q= 3  the  Tsallis  entropy  is  sensitive  enough  to  distinguish  between 
signals  with  different  probability  distribution  function  and  to  differentiate 
the  baseline  EEG  from  early  recovery  EEG. 


HI.  SIMULATIONS 

We  explore  the  use  of  parameterized  Tsallis  entropy 
(function  of  q)  to  monitor  the  changes  of  EEG  undergoes  in 
different  conditions  and  to  compare  it  with  the  entropy  of 
signals  with  known  probability  distribution  function  (PDF). 
In  particular,  we  compare  the  Tsallis  and  Shannon  entropy 
measures  on  a  synthetic  signal.  The  synthetic  signal  s(k)  is 
composed  of  four  parts  of  2000  samples  each  with  different 
PDF's:  (I)  Gaussian  distribution  with  zero  mean  and 
standard  deviation  a  =  1,  (II)  a  uniform  distribution  in  the 
interval  [-2,2],  (III)  an  EEG  recording  from  baseline,  and 
(IV)  an  EEG  recording  from  early  recovery  after 
ischemic/cardiac  arrest  brain  injury  [13].  Figure  1,  from  top 
to  bottom,  shows  one  realization  of  signal  iff)  and  Shannon 
and  Tsallis  entropies  (q=  1.5,  3  and  5)  after  50  runs. 

The  following  points  are  of  importance: 

1)  The  entropy  is  maximum  for  uniform 
distribution.  The  entropy  of  baseline  EEG  lies 
between  the  two  extreme  distributions;  Gaussian 
and  uniform,  preserving  a  subgaussian 
distribution. 

2)  Transition  from  one  distribution  to  the  next  is 
properly  revealed. 

3)  Shannon  and  Tsallis  entropy  for  q=  1 .5  are  more 
sensitive  than  Tsallis  entropy  for  q= 3  and  5  for 
segmenting  the  different  parts  of  the  signal. 

4)  Tsallis  entropy  for  q=5  has  greater  discriminant 
power  between  parts  I, II, III  and  the  part  of  the 
signal  from  recovery  EEG  (IV). 

IV.  CONCLUSIONS 

We  have  constructed  the  nonadditive  conditional  entropy 
based  on  the  axiomatic  foundations  of  classical  information 
theory  and  the  pseudoadditivity  of  the  Tsallis  entropy. 
Furthermore  we  have  shown  examples  where  the 
nonadditive  entropy  provides  a  novel  statistical  description 
of  the  brain  rhythms  during  asphyxic  injury  and  recovery. 
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